Beyond Vector Search Knowledge Management with Generative AI
Beyond Vector Search: Knowledge Management with Generative AI
Sidebar menu
Get unlimited access to the best of Medium for less than $1/week.
**Beyond Vector Search: Knowledge Management with Generative AI**
Follow
15 min read
·
Sep 3, 2023
Share
More
Press enter or click to view image in full size
Library Science is seriously underrated. You probably should hire a librarian to help you out with AI.
**Introduction**
Many companies today have begun experimenting with conversational AI tools and chatbots that are powered by large language models like GPT-3. At first, these tools seem extremely promising — the AI is able to respond coherently to prompts and appears quite intelligent. However, most teams soon start to encounter severe limitations around the AI’s inability to maintain context, lack of grounding in facts, difficulty integrating into real workflows and business logic, and challenges steering the AI away from unproductive tangents.
This article aims to provide a comprehensive overview of knowledge management strategies and concepts that can help take AI assistants to the next level. Drawing on extensive professional experience in enterprise IT infrastructure and automation as well as AI consulting, coupled with an understanding of key principles from library and information science, philosophy, and computer science, I will cover critical ideas like data ontologies, reconciliation, factual grounding, and more. The goal is to provide readers with a toolkit to transform a basic chatbot into a sophisticated AI assistant properly integrated into the company’s knowledge ecosystem, avoiding the “sad little lost robot” syndrome many teams currently face.
**The Problem: Limitations of Current AI Tools**
Early experiments using chatbots and other tools powered by large language models often seem to go well at first. The AI is able to respond intelligently to one-off prompts and appears coherent. However, real limitations quickly become apparent during sustained use.
The AI has no memory or ability to manage conversational context across multiple questions. It lacks any cumulative knowledge. The AI also has no grounding in facts or connection to authoritative sources, so it tends to hallucinate or speculate incorrectly. Another challenge is the difficulty integrating the AI into real business workflows, data sources, and logic. There is no way to smoothly interconnect the AI with other systems. Finally, it is hard to direct the focus of the AI or steer it towards valuable responses and away from unproductive tangents that waste time.
In essence, while large language models like ChatGPT demonstrate impressive capability, the AI is operating in a vacuum without the knowledge infrastructure required to serve as a truly useful assistant. Building that infrastructure is the focus of the rest of this article.
The solution? You need Knowledge Management.
**Background**
I come to this topic with over 15 years of experience as an enterprise IT infrastructure engineer focused on areas like automation, cloud computing, and virtualization. I now consult with companies specifically looking to develop and integrate AI technology into their organizations. This experience across the technical spectrum of IT gives me a rather unique perspective on how to approach developing AI assistants. Additionally, I have the benefit of being married to a professional librarian, an expert in organizing and delivering knowledge in ways that end users can easily access.
My wife has now transitioned to data product owner, so the article that follows is basically dinner table conversation for us.
**How to Use This Article**
In the following sections, I will be covering concepts drawn from diverse disciplines including information science, philosophy, computer science, and more. To get the most value, readers should aim to fully wrap their heads around these ideas, read sections multiple times if needed, look up unfamiliar terms, and synthesize how the concepts can be integrated together into product design, data management, and other aspects of an AI assistant architecture. Having the slide deck handy as a reference is highly recommended. I’m also happy to help explain concepts in more depth via my Patreon for those who need additional support. Let’s dive in!
**Key Concepts**
# **Data Ontologies**
In the realm of databases and philosophy, an ontology refers to the structured framework used to organize and define relationships between different types of data within a domain. “Ontology” basically means “nature of being.” In other words, how does the data exist?
Data ontologies refer to the structured framework used to organize and define relationships between different types of data within a domain. For example, at a hospital, the data ontology would specify that a “patient” entity has attributes like name, date of birth, address, medical history, current medications, etc. A “doctor” entity would have attributes like name, employee ID, specialty, list of patients. Relationships might include “doctor treats patient” or “patient prescribed medication”. The ontology outlines the hierarchy of broader concepts like “person” versus specific concepts like “patient” or “doctor”. It also encapsulates rules like only allowing integers for fields like age, or ensuring a patient can only have one primary physician. A clear ontology provides meaning and actionability to data that would otherwise just be an abstract structure.
Understanding data ontologies will help teams properly structure and relate the information that AI assistants require.
# **Reconciliation & Validation**
Reconciliation involves ensuring that different sets of data are consistent and accurate when combined or compared, a process often used during data integration or data migration initiatives. Key aspects include:
Reconciliation involves ensuring that different sets of data are consistent and accurate when combined or compared, a process often used during data integration projects. For example, finance teams regularly reconcile expense reports and credit card statements to ensure the company’s financial records are correct. This involves steps like data matching to align charges on the statement with their matching expense reports, error detection to find any inconsistencies, data cleansing to fix the errors, and validation against the company’s accounting system as the source of truth. The goal is to prevent bad data from propagating through the system. Thorough reconciliation is critical for AI systems as well to prevent them from operating on incorrect or contradictory information that leads to nonsensical outputs.
Thorough reconciliation is critical for preventing AI assistants from working off of bad data.
# **Factual Grounding**
Factual grounding refers to the practice of basing statements, theories, recommendations, and conclusions on verifiable facts and empirical evidence. For example, a financial advisor should ground their investment recommendations in detailed analysis of historical market data, financial filings, and economic indicators rather than speculation or personal opinion. Key elements include relying on quantifiable observed data, confirming validity by cross-referencing multiple reliable sources, ensuring logical consistency, and subjecting findings to expert scrutiny. Factual grounding prevents an AI system from generating fictional or hallucinated content by anchoring its outputs to documented facts and evidence. Just like a human analyst, an AI should cite sources and empirical data to back up any assertions.
Grounding AI in solid facts rather than speculation is crucial for keeping responses credible and avoiding hallucinated or fictitious output.
# **Source of Truth**
A source of truth refers to the authoritative, master data source that is considered the most accurate and reliable for a particular piece of information. Key attributes include:
A source of truth refers to the authoritative, master data source considered the most accurate and reliable for a particular piece of information. For example, in enterprise IT, the central authentication server acts as the source of truth for user credentials and access controls rather than having this information stored locally across individual servers. This provides consistency and prevents security issues. Another example is time servers — the National Institute of Standards and Technology (NIST) maintains atomic clocks synchronized to UTC and operates reference time servers that provide the authoritative source of truth for current time that all other systems sync to. Identifying and connecting to master authoritative sources prevents proliferation of incorrect facts and ensures systems have access to the most accurate, trusted information.
Sources of Truth can be internal to your company, or external. Identifying and connecting AI assistants to authoritative sources of truth prevents proliferation of incorrect facts across the organization.
# **Axiomatic Principles**
Axiomatic principles refer to foundational statements, assertions, or business rules that are taken to be self-evidently true. They serve as the underpinning for logical reasoning and knowledge representation. Properties include:
Axiomatic principles refer to foundational statements, assertions, or business rules that are taken to be self-evidently true. They serve as the basis for reasoning and knowledge representation. For instance, the principle of gravity does not need to be derived or proven — its universality is assumed. Similarly, organizations rely on implicit axioms, like legal compliance or ethical norms, that frame decisions and outputs. In medicine, “first, do no harm” provides an axiomatic ethical foundation. For AI systems, specifying key axioms explicitly rather than assuming them helps define the inferential basis to derive recommendations and conclusions. Articulating principles like “achieve the best possible health outcome for the patient” helps steer chatbots away from nonsensical tangents and keeps them laser focused on their mission. Stating medical ethics prevents AIs from suggesting dangerous actions, even if they seem logically valid otherwise. Making axioms explicit codifies organizational constraints.
Articulating key axioms makes it possible to embed organizational rules and constraints into the knowledge framework of an AI assistant.
Press enter or click to view image in full size
You probably don’t need to tell your chatbot about gravity, but there may be plenty of other axioms that are salient to your business or process that could be helpful.
# **Data Taxonomy**
A data taxonomy refers to the hierarchical classification and organization of data into categories, subcategories, and metadata tags. Taxonomies help structure and manage large information sets. Common uses include:
A data taxonomy refers to the hierarchical classification and organization of data into categories, subcategories, and metadata tags. For example, Carl Linnaeus created a biological taxonomy that categorized all life on earth into a tree structure starting with broad kingdoms and progressively narrowing into more specific classes, orders, families, genera, and species. This provided a conceptual framework reflecting empirical observations. Applying taxonomies to business data similarly involves establishing high-level categories under which elements can be logically grouped, then organizing the data into more granular subcategories enriched with descriptive metadata tags. As with Linnaeus’ taxonomy of life, thoughtful organization of corporate data into standardized, hierarchical taxonomies allows AI systems to more easily understand relationships and retrieve situationally relevant information.
Well-designed taxonomies allow AI assistants to better understand relationships within datasets.
# **Classification Systems**
Classification systems like the Dewey Decimal System provide standard categorical structures, notation, and naming conventions to facilitate organized storage and navigation of large data repositories. Key advantages include:
Classification systems like the Dewey Decimal System and Library of Congress provide standard categorical structures and naming conventions to facilitate organized information storage and discovery. The Dewey Decimal System classifies resources into ten broad categories, each with ten divisions and ten sections for hierarchical organization. Library of Congress builds on this with deeper layers of sub-categories. Applying similar conventions to metadata tags on data resources enables AI systems to efficiently locate and correlate information. Classification provides an orderly taxonomy for categorizing and labeling data, adhering to controlled vocabularies and notations. By indexing data elements under standardized classifications, AIs can readily cross-reference and surface relevant information even when querying unfamiliar domains. Established conventions improve coherence and navigability.
Applying classification systems helps AI agents efficiently retrieve and correlate relevant information.
# **Data Curation**
In the domain of information and library science, data curation refers to the set of activities involved in managing, preserving, enhancing, and adding value to data over time to keep it findable, accessible, and usable. Some key aspects include:
In the domain of library science, data curation refers to the activities involved in managing, preserving, enhancing, and adding value to information resources over time. Just as a library carefully builds collections, archives materials, organizes according to categorization schemes, and adds metadata for findability, curating enterprise data involves thoughtful aggregation, documentation, validation, access controls, and strategies for long-term maintenance. Diligent curation ensures corporate data remains discoverable, coherent, and usable for AI systems. Decisions regarding what raw data to ingest, what to extract into structured databases, what indices to create, and what taxonomies to apply enable AI assistants to effectively serve information needs. Data is an asset that appreciates with proper curatorial care.
Thoughtful curation of corporate data enhances relevance for AI applications while also improving longevity.
# **ETL Processes**
ETL stands for Extract, Transform, Load. It refers to a key process used in data integration to move data from assorted sources into a single, unified target repository or data warehouse. Typical steps include:
ETL stands for extract, transform, load — a key process used in data integration to move information from diverse sources into a unified data repository or warehouse. This is analogous to the logistics of a construction site, where raw materials are extracted from vendors, transformed into parts and components, then loaded onto trucks that deliver them to the appropriate workstations. In data workflows, ETL involves extracting data from various systems, transforming it to match the schema and semantics of the destination, then loading it into the target database or warehouse. Just as logistics coordinates efficient movement of physical materials between stations, ETL orchestrates safe transport of information across digital systems. Proper data handling procedures during each phase of the pipeline preserve integrity as inputs flow into AI modeling tools.
ETL workflows bring together disparate data sources into unified datasets to feed AI systems.
Press enter or click to view image in full size
Think of ETL like moving materials around a job site. The final product will be a nice fancy building, but the process is incredibly messy.
# **Information Foraging**
Information foraging refers to the behaviors and strategies humans use to seek out, gather, and consume information as needed. This is analogous to the way animals forage for food in the wild. Key aspects include:
Information foraging refers to the behaviors and strategies used to seek out and consume information as needed, much as ancient hunter-gatherers foraged for sustenance. People forage for information by leveraging contextual cues to help navigate towards sources likely to fulfill a need, while making cost-benefit tradeoffs around time and effort. An AI assistant also needs abilities to intelligently explore documents, databases, knowledge bases, and other digital hunting grounds in search of situationally relevant information. Just as hunter-gatherers develop extensive environmental knowledge about where to find certain prey or forage, AIs should build maps and models identifying where different information resources reside throughout an organization’s knowledge ecosystem. Information foraging enables AI systems to dynamically adapt as needs change.
AI requires abilities to intelligently forage through data repositories and systems for situationally relevant information.
# **Information Needs**
In library science, information needs refer to the specific types of information users require in order to effectively carry out tasks, make decisions, or solve problems. Understanding needs improves assistance. Areas of focus include:
In library science, information needs refer to the specific information required by users to carry out tasks, make decisions, or solve problems. A patron might approach a librarian seeking sources on zoning laws to assess a property, but their true information need is to determine what they can legally build. The librarian helps clarify and translate this into needs for city ordinances, permitting processes, and construction regulations. Understanding user needs allows librarians to connect patrons to the most relevant resources. Similarly, AI assistants must deeply analyze user goals and contexts to infer underlying needs, rather than just responding literally. Just as a librarian might educate a patron on effective research strategies, an AI assistant should aim to fulfill user needs from their perspective, which may require probing to determine the actual need.
Analyzing information needs provides clarity for architecting AI solutions.
**Implementation Guidelines**
Now that we have thoroughly explored the key concepts, let’s dive deeper into practical strategies and examples for implementing these principles to create a knowledge-driven AI assistant.
# **Adopt a Data-Centric Mindset**
Taking a data-centric approach means fundamentally shifting perspective to view all business activities primarily through the lens of information flows. Rather than focusing on business processes, products, or organizational structures, a data-centric mindset involves constantly examining the core information elements that ultimately drive everything.
For any given process, initiative, or task, the data-centric approach suggests starting by mapping out what information needs drive each step, what sources inputs come from, what transformations occur, and where outputs are directed. For example, a customer service inquiry starts with information input from the customer through a website form. It moves through various stations where additional context gets added, the issue categorized, knowledge applied, and a response recommended. Finally, the culmination is output back to the customer in the form of a resolution email.
Press enter or click to view image in full size
“Data is the new oil” — Information, thy god.
At its essence, this perspective recognizes that information permeates everything modern organizations do. Data is the true new oil and fuel that drives revenue, no matter what the business. With this paradigm shift, AI simply becomes a powerful new tool to harness in working with corporate information flows, rather than an end unto itself. And because future techniques will enable ever more value to be extracted from data, retaining all information for potential later re-use becomes a best practice, even if current applications are not immediately obvious.
Additionally, it is helpful to categorize AI capabilities into a few major data transformation archetypes, namely summarization, translation to new formats or languages, and expansion through generation of new content. With this taxonomy in mind, specific AI techniques can be more systematically mapped to business needs around consuming, distilling, and producing knowledge.
# **Employ Multiple Complementary Search Strategies**
Vector search technology enabled many of the initial breakthroughs in neural networks powering AI assistants. However, relying solely on vector search has inherent limitations. The most common pitfall is expecting vector search to function analogously to database query languages, when in reality it is optimized primarily for clustering semantically similar groups of text documents and extracting similarities.
Combining vector search capabilities with complementary knowledge retrieval strategies helps develop a more complete solution. For example, knowledge graphs explicitly capture relationships between entities, allowing an AI to traverse chains of facts and derive new inferences. Liberal application of descriptive metadata tags on documents provides dimensions that can be used to filter vector search results, improving relevance. For quickly locating specific pieces of information, indexes and tables of contents can provide clean hierarchies for an AI to briskly browse and select the most applicable documents. And for directly mapping user queries to precise answers, special purpose semantic matching algorithms tuned to company data can outperform generic vector search.
By blending vector search with knowledge graphs, metadata filters, content indexes, and custom search algorithms, the strengths of each approach can be leveraged while minimizing individual weaknesses.
# **Standardize Information Retrieval with Gated Processing**
Rather than treat information retrieval as a monolithic activity, dividing it into smaller modular phases with clear inputs and outputs provides more structure for an AI. For example, a three-stage gated process could entail:
First, an initial information query validation step where requirements get clearly defined, sources identified, and context established to ensure the request is coherent and answerable given available data.
Second, a distillation and extraction stage where search results get compiled, compared, and culled to extract only the most salient content needed to fulfill the specific need. Discarding superfluous information promotes precision.
Press enter or click to view image in full size
Your brain uses gates and checkpoints as your solving any problem, including information retrieval and implementation. Bring your implicit gates to explicit implementation.
Third and finally, a formatting and delivery stage where the distilled knowledge gets packaged into the required shape and shipped to the appropriate destination to complete the transaction.
By separating the knowledge acquisition process into discrete steps with asynchronous hand-offs, AI logic can be independently optimized for each stage. And instituting formal validation gates between phases reduces propagation of errors due to faulty assumptions or inadequate diligence.
# **Model Business Functions as Digital Assembly Lines**
The manufacturing concept of an assembly line, where inputs flow through a series of stations applying incremental transformations, can be a useful paradigm for architecting AI-enabled business systems. Step-by-step workflows with clear hand-offs facilitate integration. Modeling core processes as digital assembly lines entails:
First, enumerating standard inputs, analogous to raw materials entering a factory. This defines where workflows commence, such as customer service requests arriving from a web form.
Next, mapping the sequence of discrete stations, each performing information processing activities on inputs before passing them down the line. Stations represent departments, roles, applications, or even external partners tied together collaboratively.
Press enter or click to view image in full size
Visualize your entire business as information pipelines in an enormous digital factory. You have inputs, stations, transformations, and outputs. But it is all fundamentally information.
Additionally, specifying the interfaces between stations ensures smooth hand-offs. Orchestration logic tracks work products and determines when they are complete and ready to move to the next station for additional processing.
Finally, articulating desired outputs codifies what final information product or decision triggers workflow completion. For customer service, it may be resolution via email.
By breaking down convoluted processes into modular steps enacted by specialized stations interconnected by structured interfaces and data flows, automation becomes more achievable. AI can be readily inserted into appropriate points to augment the workflow.
**Conclusion**
While chatbot AI capabilities provide a promising starting point, achieving business impact requires addressing inherent limitations around contextual understanding, factual grounding, and seamless integration with existing systems and data. Blending concepts from information science, philosophy, and technology lays the groundwork. The recommendations presented aim to equip readers with both strategic perspectives and tactics to curate enterprise knowledge so that AI assistants generate maximum value, evolving from intriguing novelties into core drivers of productivity and competitive advantage. Please reach out for help applying these principles to your specific use case!
724
11
724
11
Follow
**Written by David Shapiro**
AI Thought Leader | YouTuber
**Responses (11)**
To respond to this story,
get the free Medium app.
`Not easy to understand, appreciate for sharing these information, but I have no ideas about how to use these information to improve my Generative AI apps.`
3
`Slightly different but a similar direction of taxonomy-based system + GenAI: ``https://medium.com/p/cb21b1c10557`
10
`Awesome work David. I am working on a similar mission and this article is a great summary of my current experiences and thoughts. This will help a lot to explain and guide others to achieve together the overall goal.`
1
See all responses
**More from David Shapiro**
Sep 22, 2024
What is “Post-Labor Economics”? A Gentle IntroductionQuick Intro
Aug 31, 2023
Sep 23, 2023
Sep 5, 2023
**Recommended from Medium**
In
by
Marina Wyss - Gratitude Driven
AI Agents: Complete CourseFrom beginner to intermediate to production.
Dec 6
In
by
The Best AI Tools for 2026If you’re going to learn a new AI tool, make sure it’s one of these
Dec 1
Aug 24
In
by
Oct 20
Dec 17
Sep 14